Support templated extra in outlets assets #54885
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hussein-awala
requested review from
amoghrajesh,
ashb and
kaxil
as code owners
hussein-awala
force-pushed
the
template_asset_extra
branch
from
c83fb13 to
220785a
Compare
hussein-awala
added
type:new-feature
Lee-W
reviewed
Aug 25, 2025
Member
Lee-W
left a comment
Lee-W
left a comment
There was a problem hiding this comment.
one question, one nit. no major issue on my end
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Lee-W
approved these changes
Aug 25, 2025
eladkal
approved these changes
Aug 25, 2025
Contributor
|
Not an issue, but a question: What's the intended use case here? |
Member
Author
The asset can be used to connect two DAGs for two different teams. The owner of the dataset (the team who owns the producer task) is responsible for providing some metadata to the consumer DAG to explain what to process and how. To simplify the answer, I can take a part of AIP-76 as an example. With this PR, the partition can be defined at runtime, since we can have backfills and runs created by external events, and the partition is not always known at parse time. SomeOperator(
task_id="write_asset",
...,
outlets=Asset(
"asset_example",
extra={
"partition": "{{ logical_date.strftime('%Y_%m_%d/%H') }}",
}
),
)Also, when the team owner of the dataset wants to change the partition pattern, there’s no need to notify the consumer side, as the dataset schema still respects the data contract. |
mangal-vairalkar
pushed a commit
to mangal-vairalkar/airflow
that referenced
this pull request
Aug 30, 2025
nothingmin
pushed a commit
to nothingmin/airflow
that referenced
this pull request
Sep 2, 2025
uranusjr
added a commit
to astronomer/airflow
that referenced
this pull request
Sep 3, 2025
This reverts commit 438b76c.
This was referenced Sep 3, 2025
uranusjr
added a commit
to astronomer/airflow
that referenced
this pull request
Sep 3, 2025
This reverts commit 438b76c.
hussein-awala
pushed a commit
that referenced
this pull request
Sep 4, 2025
RoyLee1224
pushed a commit
to RoyLee1224/airflow
that referenced
this pull request
Sep 8, 2025
…ache#55199) This reverts commit 438b76c.
abdulrahman305 bot
pushed a commit
to abdulrahman305/airflow
that referenced
this pull request
Sep 30, 2025
…ache#55199) This reverts commit 438b76c.
abdulrahman305 bot
pushed a commit
to abdulrahman305/airflow
that referenced
this pull request
Oct 1, 2025
…ache#55199) This reverts commit 438b76c.
abdulrahman305 bot
pushed a commit
to abdulrahman305/airflow
that referenced
this pull request
Oct 2, 2025
…ache#55199) This reverts commit 438b76c.
abdulrahman305 bot
pushed a commit
to abdulrahman305/airflow
that referenced
this pull request
Oct 3, 2025
…ache#55199) This reverts commit 438b76c.
abdulrahman305 bot
pushed a commit
to abdulrahman305/airflow
that referenced
this pull request
Oct 4, 2025
…ache#55199) This reverts commit 438b76c.
abdulrahman305 bot
pushed a commit
to abdulrahman305/airflow
that referenced
this pull request
Oct 5, 2025
…ache#55199) This reverts commit 438b76c.
abdulrahman305 bot
pushed a commit
to abdulrahman305/airflow
that referenced
this pull request
Oct 5, 2025
…ache#55199) This reverts commit 438b76c.
abdulrahman305 bot
pushed a commit
to abdulrahman305/airflow
that referenced
this pull request
Oct 7, 2025
…ache#55199) This reverts commit 438b76c.
abdulrahman305 bot
pushed a commit
to abdulrahman305/airflow
that referenced
this pull request
Oct 8, 2025
…ache#55199) This reverts commit 438b76c.
abdulrahman305 bot
pushed a commit
to abdulrahman305/airflow
that referenced
this pull request
Oct 9, 2025
…ache#55199) This reverts commit 438b76c.
abdulrahman305 bot
pushed a commit
to abdulrahman305/airflow
that referenced
this pull request
Oct 10, 2025
…ache#55199) This reverts commit 438b76c.
abdulrahman305 bot
pushed a commit
to abdulrahman305/airflow
that referenced
this pull request
Oct 11, 2025
…ache#55199) This reverts commit 438b76c.
abdulrahman305 bot
pushed a commit
to abdulrahman305/airflow
that referenced
this pull request
Oct 12, 2025
…ache#55199) This reverts commit 438b76c.
abdulrahman305 bot
pushed a commit
to abdulrahman305/airflow
that referenced
this pull request
Oct 14, 2025
…ache#55199) This reverts commit 438b76c.
abdulrahman305 bot
pushed a commit
to abdulrahman305/airflow
that referenced
this pull request
Oct 15, 2025
…ache#55199) This reverts commit 438b76c.
abdulrahman305 bot
pushed a commit
to abdulrahman305/airflow
that referenced
this pull request
Oct 17, 2025
…ache#55199) This reverts commit 438b76c.
abdulrahman305 bot
pushed a commit
to abdulrahman305/airflow
that referenced
this pull request
Oct 19, 2025
…ache#55199) This reverts commit 438b76c.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds support for Jinja templates in outlet assets, which will be rendered in the emitted events.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.